This invention relates generally to a database system, and more particularly to accessing a database table within a database system.
Database solutions that use disk storage are built on a principle that all data is stored on a disk system, and that parts of the data can be cached to computer memory for faster performance. In-memory database solutions are built on a principle that all of the data is stored on computer memory, to ensure fast access to the data. In these solutions, the in-memory data can often additionally be written or backed-up to a disk for data persistency reasons. Typical database solutions in the current marketplace use one of these two approaches.
There are several mechanisms to improve database response times such as the use of extensive buffer pooling (caching), processing as much of the data in main memory as possible, using regular disk-based algorithms, and/or making the system less vulnerable to disk operations by using high performance disks (such as solid state drives or “SSDs”). Even though these mechanisms can improve performance, they do not use leveraging algorithms that are optimized for in-memory processing even when all the data is processed in-memory. When processing data inside the main memory, using memory-optimized algorithms can lead to a significant improvement in performance. Buffer pooling mechanisms enable moving of data between a buffer pool and a disk transparently to an application, but the data in the buffer pool cannot be processed using algorithms that are optimized for in-memory usage because of a disk-optimized data layout, for example.
There are also hybrid database solutions that include both an in-memory database technology and disk database technology. In these hybrid solutions, each database table is defined either as in-memory table (m-table), or as an on-disk database table (d-table), forcing the database users to choose either of the two approaches for each table in the database schema. The division is static; rows are not transferred from the m-table to the d-table, and vice-versa, without explicitly using a transaction to insert to one table and delete from the other.
In hybrid database products or architectures having several database servers it is possible to programmatically (at an application level) store some data into an in-memory database server and other data in a disk-based server. Controlling this data placement on the application level is, however, extremely tedious and complicated and increases the vulnerability of the system and may compromise data integrity. Additionally, a challenge with in-memory databases and database tables is that when a database table grows large enough, it cannot be stored in the in-memory database any longer due to lack of available memory. In general, databases tend to grow over time for multiple reasons, and in-memory database tables have a hard limit in terms of the maximum size of the available memory.
One solution for addressing this memory database and database table growth problem is to use a hybrid database table, where some of the rows are handled by way of the in-memory database technology, and some of the rows are handled by way of the disk database technology (i.e., similar to a hybrid database solution, but within one table). A hybrid table keeps all the data logically in the same database table, but the data is physically divided between an in-memory part (m-part), and a disk part (d-part). The paper “Hybrid In-Memory and On-Disk Tables for Speeding-Up Table Access” by Guisado-Gámez et. al, published in Database and Expert Systems Applications, Lecture Notes in Computer Science, 2010, Volume 6261/2011, p.231-240 describes such a hybrid solution. One of the problems with contemporary hybrid tables is the index structure for accessing the data, since in-memory database tables typically have different index structures than disk database tables.